Papers with transformer model
English to Hindi Multi-modal Neural Machine Translation and Hindi Image Captioning (D19-52)
Copied to clipboard
| Challenge: | Multi-modal translation is an emerging task of the MT community, where visual features of image combine with textual features of parallel source-target text to translate sentences. |
| Approach: | They propose to use convolutional neural net-works and visual geometry to extract image features and attention-based Neural MachineTranslation (NMT) system for translation. |
| Outcome: | The proposed multi-modal translation system improves translation quality and improves the quality of the captions of the images. |
The Concordia NLG Surface Realizer at SRST 2019 (D19-63)
Copied to clipboard
| Challenge: | The goal of Natural Language Generation (NLG) is to produce natural texts given structured data. |
| Approach: | They propose a model for the shallow track of the 2019 NLG Surface Realization Shared Task . they divided the problem into two sub-problems: reordering and inflecting . |
| Outcome: | The proposed model reconstructs sentences whose word order and word inflections were removed. |
Label efficient semi-supervised conversational intent classification (2023.acl-industry)
Copied to clipboard
| Challenge: | A conversational chatbot can answer pre-purchase questions and post-purchase queries to provide a seamless shopping experience. |
| Approach: | They propose a semi-supervised learning approach for label-efficient intent classification using a small labeled corpus and large unlabeled query data to train a transformer model. |
| Outcome: | The proposed approach significantly improves over the baseline, even with a limited labeled set. |
Low-resource neural machine translation with morphological modeling (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for character-based and sub-word tokenization are limited to the surface forms of the words. |
| Approach: | They propose a framework-solution for modeling complex morphology in low-resource settings using a transformer architecture and beam search-based decoder. |
| Outcome: | The proposed model improves translation performance on Kinyarwanda English translation using public-domain parallel text. |
Transformer and seq2seq model for Paraphrase Generation (D19-56)
Copied to clipboard
| Challenge: | Existing methods for generating paraphrases fall into one of these broad categories -rule-based, seq2seq, deep generative models and a varied combination. |
| Approach: | They propose a framework that combines transformer and sequence-to-sequence models for better quality of generated paraphrases. |
| Outcome: | The proposed framework improves on two datasets-QUORA and MSCOCO using transformer and sequence-to-sequence models. |
Towards Large-Scale Interpretable Knowledge Graph Reasoning for Dialogue Systems (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing systems that require extensive labor to process user requests are limited in their reasoning capabilities and require extensive manual effort to design. |
| Approach: | They propose a method that allows a transformer model to walk on a large-scale knowledge graph to generate responses by reasoning over differentiable knowledge graphs. |
| Outcome: | The proposed method allows a transformer model to walk on a large-scale knowledge graph to generate responses. |
Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Tasks (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for supervised meta-learning require many training tasks to generalize . cloze-style objectives can be used to generate a large, rich, meta-training task distribution from unlabeled text. |
| Approach: | They propose a self-supervised approach to generate a large, rich, meta-learning task distribution from unlabeled text. |
| Outcome: | The proposed approach generates a large, rich, meta-learning task distribution from unlabeled text. |
CHOLAN: A Modular Approach for Neural Entity Linking on Wikipedia and Wikidata (2021.eacl-main)
Copied to clipboard
Manoj Prabhakar Kannan Ravi, Kuldeep Singh, Isaiah Onando Mulang’, Saeedeh Shekarpour, Johannes Hoffart, Jens Lehmann
| Challenge: | Existing approaches to target end-to-end entity linking over knowledge bases are not efficient. |
| Approach: | They propose a modular approach to target end-to-end entity linking over knowledge bases. |
| Outcome: | The proposed approach outperforms state-of-the-art approaches on two well-known knowledge bases. |
Generative Dictionary: Improving Language Learner Understanding with Contextual Definitions (2024.emnlp-demo)
Copied to clipboard
| Challenge: | GenerativeDictionary generates word sense interpretations based on context . traditional word sense disambiguation methods may not capture the intended word sense . |
| Approach: | They propose a dictionary system that generates word sense interpretations based on context . they transform context sentences to highlight the meaning of target words . |
| Outcome: | The proposed dictionary system is comparable to traditional word sense disambiguation methods. |
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021.acl-long)
Copied to clipboard
| Challenge: | State-of-the-art parameter-efficient fine-tuning methods rely on introducing adapter modules between the layers of a pretrained language model. |
| Approach: | They propose a framework that can learn adapter parameters for all layers and tasks by generating them using shared hypernetworks. |
| Outcome: | The proposed framework improves performance on the well-known GLUE benchmark while adding only 0.29% parameters per task. |
On Orthogonality Constraints for Transformers (2021.acl-short)
Copied to clipboard
Aston Zhang, Alvin Chan, Yi Tay, Jie Fu, Shuohang Wang, Shuai Zhang, Huajie Shao, Shuochao Yao, Roy Ka-Wei Lee
| Challenge: | a dedicated study on orthogonality constraints for transformers has been lacking . plug-and-play constraints increase the BLEU of transformers . |
| Approach: | They propose to use plug-and-play constraints to encourage matrices to be orthogonal for numerical stability. |
| Outcome: | The proposed constraint increases the BLEU on the large-scale WMT’16 EnDe benchmark by a factor of 28.4 to 29.6. |
Vector Space Interpolation for Query Expansion (2022.aacl-short)
Copied to clipboard
| Challenge: | Topic-sensitive query set expansion is crucial for queries related to sensitive and emerging topics. |
| Approach: | They propose a method for topic-sensitive query set expansion using vector space interpolation. |
| Outcome: | The proposed method generates new queries about the sensitive topic by incorporating set diversity, which is not captured by traditional sentence-level augmentation methods such as paraphrasing or back-translation. |
Cross-Lingual Transfer in Zero-Shot Cross-Language Entity Linking (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing work on cross-language entity linking grounds mentions written in multiple languages to a monolingual knowledge base is lacking. |
| Approach: | They propose a task that uses multilingual BERT representations of both the mention and context as input and explore zero-shot language transfer. |
| Outcome: | The proposed model performs well in both monolingual and multilingual settings. |
Conversational Question Answering over Knowledge Graphs with Transformer and Graph Attention Networks (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing knowledge graphs are widely used for (complex) conversational question answering . LASAGNE improves the F1-score on eight out of ten question types . |
| Approach: | They propose a multi-task neural semantic parsing approach for (complex) conversational question answering over a knowledge graph using a transformer model and a Graph Attention Networks model. |
| Outcome: | The proposed approach outperforms baselines on eight out of ten question types on a standard dataset for complex sequential question answering. |
Hengam: An Adversarially Trained Transformer for Persian Temporal Tagging (2022.aacl-main)
Copied to clipboard
| Challenge: | A wide array of natural language processing (NLP) applications relies on accurately identifying events and their respective occurrence times. |
| Approach: | They propose an adversarially trained transformer for Persian temporal tagging that can generalize over the HengamTagger’s rules. |
| Outcome: | The proposed tool outperforms state-of-the-art methods on a diverse and manually created dataset. |
How do we get there? Evaluating transformer neural networks as cognitive models for English past tense inflection (2022.aacl-main)
Copied to clipboard
| Challenge: | English past tense inflections is a typical quasi-regularity task, but it is criticized that it learns only to generalize the most frequent pattern, but not the regular pattern. |
| Approach: | They train a set of transformer models with different settings to examine their behavior on a typical English quasi-regularity task. |
| Outcome: | The models achieved high accuracy on unseen regular verbs and some accuracy on unseen irregular verbs. |
CNNBiF: CNN-based Bigram Features for Named Entity Recognition (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Named entity recognition tasks require a self-attention mechanism with unconstrained length that fails to capture local dependencies. |
| Approach: | They propose a joint training objective which better captures the semantics of words corresponding to the same entity by augmenting the objective with a group-consistency loss component. |
| Outcome: | The proposed model achieves a test F1 of 93.98 with a single transformer model. |
Narrative Embedding: Re-Contextualization Through Attention (2021.emnlp-main)
Copied to clipboard
| Challenge: | a novel approach to narrative event representation uses attention to re-contextualize events across the whole story . a recent study shows that attention is used to attach event semantics to tokens . |
| Approach: | They propose an unsupervised approach to narrative event representation using attention to re-contextualize events across the whole story. |
| Outcome: | The proposed approach achieves state of the art performance on multiple choice and story cloze tasks. |
SQUIRE: A Sequence-to-sequence Framework for Multi-hop Knowledge Graph Reasoning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for multi-hop knowledge graph reasoning suffer from slow and poor convergence . a transformer model can be used to learn and predict in an end-to-end fashion, giving faster convergence compared to previous methods . |
| Approach: | They propose a Sequence-to-sequence based multi-hop reasoning framework . it uses an encoder-decoder transformer structure to translate the query to a path . |
| Outcome: | The proposed framework can learn and predict in an end-to-end fashion, which gives better and faster convergence. |
SAFR: Neuron Redistribution for Interpretability (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies on controlling neuron distribution for interpretability have focused on focusing on monosemanticity instead of focusing solely on feature interactions. |
| Approach: | They propose a method to regularize feature superposition by encoding representations of multiple features within a single neuron. |
| Outcome: | The proposed method improves model interpretability without compromising prediction performance. |
Temporal Attention for Language Models (2022.findings-naacl)
Copied to clipboard
| Challenge: | Pretrained language models are trained on corpora derived from the web, but ignore this information. |
| Approach: | They propose a time-aware self-attention mechanism that captures time-specific contextualized word representations and allows the transformer to capture this information. |
| Outcome: | The proposed model achieves state-of-the-art on three datasets in different languages (English, German, and Latin) that vary in time, size, and genre. |
On the Helpfulness of Document Context to Sentence Simplification (2020.coling-main)
Copied to clipboard
| Challenge: | Text simplification is a hot issue in the field of natural language generation (NLG). |
| Approach: | They propose to use Wikipedia context to improve sentence simplification by using neural networks to learn the effects of preceding and following sentences on current sentences. |
| Outcome: | The proposed model outperforms the best performing model on the baseline dataset by 2.46 (7.22%). |
Multilingual Neural Machine Translation with Deep Encoder and Multiple Shallow Decoders (2021.eacl-main)
Copied to clipboard
| Challenge: | Recent work in multilingual translation has improved translation quality surpassing bilingual baselines using deep transformer models with increased capacity. |
| Approach: | They propose a deep encoder with multiple shallow decoders to reduce inference latency while maintaining translation quality. |
| Outcome: | The proposed model achieves 1.8x speedup on average compared to a standard transformer model with no drop in translation quality. |
Character-Level Translation with Self-attention (2020.acl-main)
Copied to clipboard
| Challenge: | Existing models for character-level neural machine translation operate on word-level, which makes them memory inefficient because of large vocabulary sizes. |
| Approach: | They propose a transformer-based model and a novel variant that uses convolutions to combine information from nearby characters to facilitate character interactions. |
| Outcome: | The proposed model outperforms the standard transformer model and learns more robust character alignments on bilingual and multilingual translation datasets. |
NarrowBERT: Accelerating Masked Language Model Pretraining and Inference (2023.acl-short)
Copied to clipboard
| Challenge: | Large-scale language model pretraining is expensive as the models and pretraining corpora have become larger over time. |
| Approach: | They propose a modified transformer encoder that increases throughput for masked language model pretraining by more than 2x. |
| Outcome: | The proposed model increases throughput on IMDB and Amazon reviews classification and CoNLL NER tasks by 3.5x with minimal performance degradation. |
BERT Prescriptions to Avoid Unwanted Headaches: A Comparison of Transformer Architectures for Adverse Drug Event Detection (2021.eacl-main)
Copied to clipboard
| Challenge: | Pretrained transformer-based models are a common choice for identifying drug events from social media texts. |
| Approach: | They propose to compare transformer-based models with in-domain language pretraining to find out which one is better at ADE detection. |
| Outcome: | The proposed models outperform SpanBERT and PubMedBERT on two benchmarks. |
Reinforced Sequence Training based Subjective Bias Correction (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing methods for subjective bias correction focus on making one-word edits . a hybrid method to improve the performance of Seq2Seq and transformer-based bias correction models is proposed . |
| Approach: | They propose a reinforced sequence training approach for robust subjective bias correction . it balances bias neutralization with fluency and semantics preservation through reinforcement learning . |
| Outcome: | The proposed method improves subjective bias correction over existing methods . it is cross-trained over multiple sources of bias to be more robust to new styles of biased writing . |
Exploiting Definitions for Frame Identification (2021.eacl-main)
Copied to clipboard
| Challenge: | a frame-semantic parsing task is to determine which frame best captures the meaning of a word or phrase in a sentence. |
| Approach: | They propose a frame identification model that generates representations for frames and lexical units (senses) they evaluate the model on three data sets and show it consistently achieves better performance than previous systems. |
| Outcome: | The proposed model consistently outperforms previous systems on three data sets. |
Syntax-guided Neural Module Distillation to Probe Compositionality in Sentence Embeddings (2023.eacl-main)
Copied to clipboard
| Challenge: | Past work on sentence embedding models faces issues determining the causal impact of implicit syntax representations. |
| Approach: | They construct a neural module net based on a transformer model and train it end-to-end to approximate the sentence’s embedding. |
| Outcome: | The proposed model captures whether syntax is a strong model of its compositional ability. |
Identifying the limits of transformers when performing model-checking with natural language (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent studies have focused on transformer models’ ability to perform reasoning on text, but the above question has not been adequately answered. |
| Approach: | They investigated the problem of model-checking with natural language to determine whether transformers can comprehend logical semantics in natural language. |
| Outcome: | The proposed model-checking problem is suited to address this issue but is untouched in natural language inference research. |
Task-Aware Representation of Sentences for Generic Text Classification (2020.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to text classification use a transformer architecture with a linear layer on top. |
| Approach: | They propose a transformer-based approach that outputs a class distribution for a given prediction problem. |
| Outcome: | The proposed model outperforms existing approaches on small training data and can learn to predict new classes even with no training examples. |
Improve Transformer Models with Better Relative Position Embeddings (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for generating position embeddings are not fully utilized in NLP tasks. |
| Approach: | They propose to generalize the absolute position embedding to a generalized relative position embedded method . they also propose to use the relative embeddable method to improve the accuracy of large models . |
| Outcome: | The proposed method improves accuracy on the SQuAD1.1 dataset compared to previous methods . it can be easily adopted as a drop-in replacement for improving accuracy of large models . |
Predicting Discourse Trees from Transformer-based Neural Summarizers (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing extractive summarization tasks use only neural approaches to learn discourse information, but recent work has shown that it is beneficial for summarizing discourse information. |
| Approach: | They propose to generate document-level discourse trees from pre-trained neural summarizers that encode dependency- and constituency-style discourse information. |
| Outcome: | The proposed model learns both, dependency- and constituency-style discourse information, consistent with pre-neural results. |
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022.emnlp-main)
Copied to clipboard
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, Xudong Shen
| Challenge: | a benchmark of 1,616 diverse NLP tasks and their expert-written instructions is used to test generalization of models to unseen tasks . a recent study shows that instruction-following models outperform instruction-based models by over 9% . |
| Approach: | They build a benchmark of 1,616 diverse NLP tasks and their expert-written instructions. |
| Outcome: | The proposed model outperforms existing instruction-following models by over 9% on the benchmark despite being smaller. |
Transformer Feed-Forward Layers Are Key-Value Memories (2021.emnlp-main)
Copied to clipboard
| Challenge: | Using feed-forward layers, we show that the learned patterns are human-interpretable, and that lower layers tend to capture shallow patterns, while upper layers learn more semantic ones. |
| Approach: | They propose that feed-forward layers in transformer-based language models operate as key-value memories where each key correlates with textual patterns in the training examples and each value induces a distribution over the output vocabulary. |
| Outcome: | The proposed model is based on key-value memories with a key-level correlation with the training examples and a distribution over the output vocabulary. |
Linguistically-Motivated Yorùbá-English Machine Translation (2022.coling-1)
Copied to clipboard
| Challenge: | Several phenomena where asymmetry arises have been identified as challenging problems for machine translation. |
| Approach: | They perform a fine-grained analysis of how an SMT system compares with two NMT systems when translating bare nouns into English. |
| Outcome: | The proposed model outperforms the SMT and BiLSTM models for 4 categories and the BiLST outperformed the SLT models for 3 categories. |
Empathetic and Emotionally Positive Conversation Systems with an Emotion-specific Query-Response Memory (2022.findings-emnlp)
Copied to clipboard
Zhiliang Tian, Yinliang Wang, Yiping Song, Chi Zhang, Dongkyu Lee, Yingxiu Zhao, Dongsheng Li, Nevin L. Zhang
| Challenge: | Existing emotional conversation systems output responses according to either a given emotion or the user’s emotion reflected in the input queries. |
| Approach: | They propose to generate empathetic responses catering to the user’s emotions while leading the conversation to be emotionally positive by abstracting the conversation corpus and extracting the different responding strategies for different users’ emotions and conversational topics into a memory. |
| Outcome: | The proposed model surpasses the baseline methods in appropriateness, diversity, and generating emotionally positive responses. |
The Cascade Transformer: an Application for Efficient Answer Sentence Selection (2020.acl-main)
Copied to clipboard
| Challenge: | Recent research shows that transformer-based neural networks can greatly advance the state of the art over many natural language processing tasks. |
| Approach: | They propose a technique to adapt transformer-based models into a cascade of rankers. |
| Outcome: | The proposed technique reduces computation by 37% with almost no impact on accuracy on two English question answering datasets. |
Dodo: Dynamic Contextual Compression for Decoder-only LMs (2024.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to NLP are sparsifying attention patterns or approximating the attention computation with kernel methods. |
| Approach: | They propose a method for dynamic contextual compression for decoder-only LMs. |
| Outcome: | The proposed method reduces the cost of self-attention to a fraction of typical time and space. |
Do transformer models do phonology like a linguist? (2023.findings-acl)
Copied to clipboard
| Challenge: | Neural sequence-to-sequence models have been very successful at tasks in phonology and morphology that require a capacity for intricate linguistic generalisations. |
| Approach: | They propose to use a transformer model to model phonological generalisations on 29 different processes . they also use phonology as a primer and a rule with a withheld case to infer behaviour . |
| Outcome: | The proposed model can model all 29 phonological phenomena on a constructed language, regardless of perceived process difficulty. |
IgboBERT Models: Building and Training Transformer Models for the Igbo Language (2022.lrec-1)
Copied to clipboard
| Challenge: | This paper focuses on building resources for named entity recognition for Igbo, a language mainly spoken in the south eastern part of Nigeria. |
| Approach: | They present a standard Igbo named entity recognition dataset and results from fine-tuning transformer IgbeNER models. |
| Outcome: | The proposed dataset and model improves on the IgboNER task while training and fine-tuning a transformer model with comparatively little Igbe text data. |
Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing transformer models are computationally demanding and prohibitively costly for long sequences due to the quadratic complexity of its selfattention module. |
| Approach: | They propose a transformer-based model that inherits weights from large pretrained models by removing redundancies in hidden sequences using the ready-made Fast Fourier Transform operator. |
| Outcome: | The proposed model outperforms the standard BART model on the long-range modeling benchmark LRA with significant improvements in speed and space. |
Standardisation of Dialect Comments in Social Networks in View of Sentiment Analysis : Case of Tunisian Dialect (2022.lrec-1)
Copied to clipboard
| Challenge: | Using the internet, the spoken Arabic dialect language becomes informal languages written in social media . this linguistic situation inhibits mutual understanding and makes computational approaches difficult . we present a pipeline to standardize the written texts in social networks by translating them to MSA . |
| Approach: | They propose a pipeline to standardize Arabic written texts by translating them to MSA . they use a bert-based model to select Tunisian Dialect from MSA and other dialects . |
| Outcome: | The proposed pipeline achieves the best score for the standardization of written texts in social networks . the proposed pipeline includes the translated TD and the original text written in MSA . |
Autoregressive Reasoning over Chains of Facts with Transformers (2020.coling-main)
Copied to clipboard
| Challenge: | Existing work on this task either evaluates facts in isolation or artificially limits the possible chains of facts, thus limiting multi-hop inference. |
| Approach: | They propose an iterative inference algorithm that decomposes the selection of facts from a corpus autoregressively and conditioning the next iteration on previously selected facts. |
| Outcome: | The proposed method outperforms the previous state-of-the-art in terms of precision, training time and inference efficiency on the WorldTree dataset. |
Human Guided Exploitation of Interpretable Attention Patterns in Summarization and Topic Segmentation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have investigated the multi-head self-attention mechanism of transformers. |
| Approach: | They propose to use a human-in-the-loop pipeline to discover task-specific attention patterns and inject them into transformer models to improve their accuracy. |
| Outcome: | The proposed methods improve the performance of transformer models by incorporating predefined patterns into their attention matrices. |
Harder Task Needs More Experts: Dynamic Routing in MoE Models (2024.acl-long)
Copied to clipboard
Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Songfang Huang, Yansong Feng
| Challenge: | Unlike existing MoE approaches that rely on fixed TopK Routing, our dynamic expert selection framework dynamically allocates experts based on the confidence level in expert selection for each input. |
| Approach: | They propose a dynamic expert selection framework that dynamically allocates experts based on the confidence level in expert selection for each input. |
| Outcome: | The proposed method achieves an average improvement of 0.7% with less than 90% activated parameters and outperforms dense models in QA and machine translation tasks. |
Choose Your Transformer: Improved Transferability Estimation of Transformer Models on Classification Tasks (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for NLP tasks require fine-tuning, but it is computationally infeasible. |
| Approach: | They propose an approach that inexpensively estimates a ranking of the expected performance of a given set of transformer language models for a specific task. |
| Outcome: | The proposed model improves the Pearson correlation coefficient between the true model ranks and the estimate. |
Seq2SeqPy: A Lightweight and Customizable Toolkit for Neural Sequence-to-Sequence Modeling (2020.lrec-1)
Copied to clipboard
| Challenge: | Neural models have attracted a lot of attention in the past few years due to their complexity and need to be customized to meet specific needs. |
| Approach: | They propose a lightweight toolkit for sequence-to-sequence modeling that prioritizes simplicity and ability to customize the standard architectures easily. |
| Outcome: | The proposed tool performs similarly or even better than a very widely used sequence-to-sequence toolkit. |
RingFormer: Rethinking Recurrent Transformer with Adaptive Level Signals (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Transformers have shown strong performance in processing sequential data, but their parameters are larger . a novel approach to reduce the model parameters while maintaining high performance is proposed . |
| Approach: | They propose a transformer-based model that processes input repeatedly in a circular, ring-like manner. |
| Outcome: | The proposed approach reduces model parameters while maintaining high performance . the proposed approach is validated in the experiments. |
Transformer-based Joint Modelling for Automatic Essay Scoring and Off-Topic Detection (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies show that automated essay scoring systems assign lower grades to irrelevant responses. |
| Approach: | They propose an unsupervised technique that jointly scores essays and detects off-topic essays. |
| Outcome: | The proposed method outperforms baseline and earlier conventional methods on two essay-scoring datasets in off-topic detection and on-topic scoring. |
Understanding How Positional Encodings Work in Transformer Model (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing studies have reported superiority of relative PEs in translation tasks. |
| Approach: | They analyze in which part of a transformer model PEs work and compare them using experiments . they find that relative PEs should be added only to query and key of attention mechanism . |
| Outcome: | The results show that relative and absolute PEs work in a transformer model, and should be added to the query and key of an attention mechanism, not to the value. |